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(54) Splicing of video data In progressively mfr^shed video streams 



(57) A method is provided for r^adng compressed 
video from a first source with vkleo from a second 
source at a predefined insertion point in a video data 
stream. If the video from the second source is progres- 
sively refreshed video, it is decompressed sufficiently 
prior to the insertion point to enabUe the recovery and 
recompression of a first video frame (in encoding order) 
from the second source to be inserted at the insertion 
point. The recovered first video frame is then reconv 
pressed as an intra-coded (I) frame. The insertion of the 



video from the second source is then commenced at the 
insertion point using the I frame. The insertion point 
can, for example, indicate the commencement of a com- 
mercial break in the video stream, in which case the 
second source provides a commercial for insertion into 
the commercial break. Where the commercial is stored 
in menfK)ry, it can be ^red with the first frame thereof 
(in ^coding order) as an I frame. 
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Description 

BACKGROUND OF THE INVENTION 

[0001 ] The present invention relates to digital televi- s 
sion, and more particularly to a method for video inser- 
tion or stream switching in progressively refreshed 
video streams. 

[0002] Commercial insertion, promo insertion, stu- 
dio routing, camera switching, tape editing, and the like io 
are basic operations in television production and broad- 
casting. In present day analog systems, switching takes 
place In the vertical interval of the video signal. With the 
advent of digital television as implemented, for example, 
in the Motion Picture Experts Group (MPEG) standards, is 
switching or "splicing" with compressed bitstreams Is 
very complicated. In order to facilitate such operations, 
the MPEG-2 standard defines "splice points" where 
point signals (e.g., a commercial) can be substituted for 
another signal (e.g.. a main program) in a video stream. 20 
[0003] The Society of Motion Picture and Television 
Engineers (SMPTE) Is currently working on a draft 
standard for splicing MPEG-2 transport streams. See, 
for example, Proposed SMPTE Standard PT20.02y010 
"Splice Points tor MPEG-2 Transport Streams". Second 2S 
Draa July 1997. incorporated herein by reference. 
[0004] There are various reasons why splicing 
MPEG video is more difficutt than switching unconv 
pressed video. In order to explain this, it is necessary to 
understand some basics about the MPEG transport 30 
stream. MPEG gets much of its compressbn capabilrty 
by saxling only the changes between different vkleo 
frames. A first type of frame, known as a prediction or 
"P" frame, is used by the decoder to predict the frame 
from a previous "P" frame or from an intra-coded "1" 3s 
frame in the video stream. The stream merely can-ies 
Tme tuning" information to correct errors from an 
approximate prediction. An I frame is conrpressed with- 
out motion predkrtion. Thus, a full frame can be recon- 
structed from an I frame vnthout reference to any other 40 
frame. In this manner, errors in the prior frame predic- 
tions will be eliminated once an I frame an^ives and is 
decoded. 

[0005] Bkiirectional "B" frames are like P frames, 
except that the prediction is made not only from the pre- 45 
vious I or P frame, but also from the next one. For pur- 
poses of splicing, this means that while it is possible to 
leave an old stream after any P or I frame, rt is not pos- 
sible to les^e the stream after a B frame since that 
wouW cut off the backward prediction reference frame so 
for that B frame. From the point of view of the bitslream. 
this means that a first video stream may only be left 
after a P or I frame and all of the B frames (if any) that 
immediately follow it have passed. 
[0006] Matters are further complicated when the ss 
conpressed MPEG bitstream is progressively 
refreshed. In progressively refreshed streanr». such as 
taught in commonly assigned US patent 5.057,916 to 



Krause et al. entitled "Method and Apparatus for 
Refreshing Motion Compensated Sequential Video 
Images." sequential video images are refreshed one 
region at a time. A different region in each of the plurality 
of vkieo images is communicated without compression 
during a refresh cycle. In this manner, an image area 
defined by the region is progressively refreshed by the 
non-compressed regions during the refresh cycle. Com- 
pression of video images may be controlled to prevent 
data contained in regions not yet refreshed during a cur- 
rent refresh cycle from corrupting data contained in 
regions that have t>een refreshed during the current 
refresh cycle. 

[0007] For commercial insertion or switching pur- 
poses, compressed video from a first source is replaced 
with compressed vkleo from a second source at a pre- 
defined insertion point in the video data stream. If the 
video from either or both of the first and second source 
is progressively refreshed, then progressive refresh 
related artifacts will appear at one or both of the bound- 
aries between the first and second source vkieo. 
[0008] It would be advantageous to provkte a 
method for providing splteing of progressively refreshed 
video streams that minimizes noticeable artifacts. The 
present invention provides such a method. 

SUMIMARY OF THE INVENTION 

[0009] In accordance with the present invention, a 
method is provkled for replacing compressed vkjeo 
from a first source wrtfi compressed video from a sec- 
ond source , e.g., at a predefined insertion point in a 
video data stream. A detemiinatfon is made as to 
whether the video from the second source is progres- 
sively refreshed video. The determination can be a real 
time determination (e.g.. based on testing the second 
source) or can result from a priori knowledge of the sec- 
ond source. If the video from the second source is pro- 
gressively refreshed, then it is decompressed 
suffKiently pria in time to the replacement of the first 
source video to enable the recovery and recompression 
of a first replacement vkieo frame from sakJ second 
source before replacement commences. It is noted that 
for purposes of present disclosure, the "first replace- 
ment vWeo frame" is the first frame In encoding order of 
the replacement vkleo stream, which may differ from the 
display order, as well known in the art. 
[0010] The recovered first video frame is reoom- 
pressed as an intra-coded (I) frame. Then, insertion of 
the vkleo from the second source is commenced (e.g., 
at the predefined insertion point) using the 1 frame. 
Since the 1 frame comprises all of the information nec- 
essary to reconstruct a full frame of data and is not pro- 
gressively refreshed, it enables a clean switch to be 
nrade from the video data from the first source to the 
video data from the second source. 
(POII] The insertion point can. for exanrple. indi- 
cate the conrunencement of a commercial break in the 
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video stream. In this instance, the second source pro- 
vides a commercial for insertion into the commercial 
break. At the conclusion of the oommercial break, video 
from the first source recommences at a second inser- 
tion point in the data stream. As with the first splk;e, it is 5 
necessary to determine if the video from the first source 
is progressively refreshed vkjeo. If so. that video is 
decompressed sufficiently prior to the second insertion 
point to enable the recovery and recompression of a 
first video frame (in encoding order) from the first source 70 
to be inserted at the second insertion point. The recov- 
ered first video frame from the first source is then rec- 
ompressed to produce an I frame. Insertion of the video 
from the first source at the second insertion point com- 
mences using the I frame produced from the first 15 
source. 

[001 2] A method is also provided for replacing com- 
pressed video from a first source with sut>stitute pro- 
gressively refreshed compressed video (e.g.. at a 
predefined insertion point) in a video data stream. The 20 
substitute video (e.g., a commercial) is stored with afirsl 
frame thereof encoded as an intra-coded (I) frame. The 
insertion of the substitute video is commenced (e.g., at 
the predefined insertion point) using the I frame as the 
first inserted frame. When the compressed video from 2S 
the first source is progressively refreshed and is to be 
recommenced at a second insertion point in tiie data 
stream following tiie first insertion point (e.g.. at the end 
of a commercial break) the video from the first source 
must be processed to prevent progressive refresh 30 
related artifacts. In particular, the video from the first 
source is decompressed sufficientiy prior to tiie second 
insertion point to enatie the recovery and recompres- 
sion of a first video frame from ttie first source to be 
inserted at the second insertion point The recovered 35 
first vkjeo frame from the first source is recompressed 
to produce an l-frame. Insertion of the video from tiie 
first source is commenced at the second insertion point 
using the I frame produced from ttie first source. 

40 

BRIEF DESCRIPTION OF THE DRAWINGS 
[0013] 

FIG. 1 is a diagram illustrating first and second data 45 
streams 81. S2. respectively, togettier witti a ttiird 
data stream S3 comprising sti-eam 81 into which a 
portion of stream 82 has been inserted; and 
FIG. 2 is a (fiagram illustrating a data stream 84 
created by switching from data stream 81 to data so 
stream 82. 

DETAILED DESCRIPTION OF THE INVENTION 



[0014] The present invention provides a method for 
inserting commercials or other video segments into pro- 
gressively refreshed bitsti-eams and for switching from 
one bitstream to another when any of the brtst^eams is 
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progressively refreshed. 

[0015] FIG. 1 illustrates a first compressed digital 
video stream 10 (stream 81). The video stream 10 has 
a first insertion point t1 at which a commercial break 
commences. The commercial break temiinates at inser- 
tion point t2. It is noted that tfie insertion points may or 
may not comprise predetermined points in the video 
stream. Since it is more typical to insert a video seg- 
ment such as a commercial at a predefined time, it is 
expected that predetermined insertion points will more 
often be used with the present invention. However, the 
use of predetermined insertion points is not required in 
accordance witii the invention. For example, a local tel- 
evision stucBo may desire to insert a local segment into 
a main video stream at an arbitrary time, in whch case 
the invention may be implemented without using a pre- 
determined insertion point for ttie replacement of tfie 
main video witti ttie replacement video. 
[001 6] As indicated in FIG. 1 , a commercial or other 
video segment Sic is inserted in the oommercial break 
defined by insertion points t1 and t2. In particular, ttie 
inserted video 81c commences at insertion point t1 and 
terminates at insertion point t2, where the main video 
program can-ied by 81 restarts at tiie temiination of ttie 
commercial break. 

[0017] A similar video stream 12 (stream 82) com- 
prises a main program 82 into which a commercial S2c 
is inserted. It is noted ttiat ttie commercials Sic and 82c 
may be part of ttie streams 81 and 82, respectively, in 
which case ttiey need not be separately inserted into 
the main stream. However, in tiie event tfiat the com- 
mercials (or ottier video segments tiiat are inserted) are 
stand alone video segments, they must be inserted into 
ttie main stream at the appropriate insertion point t1 and 
terminated at ttie insertion point t2 when tiie main 
stream recommences. 

[0018] Video stream 14 (stream 83) represents a 
classic case of video splicing, in which part of stream SI 
is replaced by 82c. As indicated above, S2c can be part 
of stream 82 or may be a stand alone commercial or 
ottier stand alone video segment. If 81 and/or 82 are 
progressively refreshed video data streams, then pro- 
gressive refresh related artifacts will occur at one or 
both the boundaries of 81 and 82c. Such refresh 
related artifacts can last as long as one second or more, 
depending upon the refresh rate. Thus, when a com- 
mercial is switched in and out one would see ttie arti- 
facts every time the switch is made. To prevent such 
artifacts while avitching the commercial into a main 
data stream (i.e.. splicing out of the main stream and 
cutting the commercial in), the commercial can be com- 
pressed in I. P. B frame nrxxle. However, when ttie com- 
mercial is over and the progressively refreshed main 
stream is switched back in. the progressive refresh arti- 
facts will still occur. It is noted ttiat the video frames in 
ttie sfream illustrated in FIG. 1 are in encoding order, 
which is different from ttie frame display order. When 
ttie frames are encoding order, ttie first frame of ttie 
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switched-in segment will be an I frame. 
[0019] For purposes of the present disclosure, the 
terms "frame" and "video frame" are used in a generic 
sense to mean a portion of a video signal or stream 
from which other portions are reconstructed using, for 
exanple, an l-frame, P-frame a B-frame. Thus, 
although typically a "frame" wifl be a lull" video frame, it 
is not intended to limit the meaning thereof to a full video 
frame for purposes of the present disclosure, since 
inplementations may exist or be developed wherein. 
e.g., subframe. field, subfield, or other processing is 
used. With respect to processing of video on afield level 
instead of a frame level, it should be appreciated that t- 
f ields, P-fields and &f ietds may be used. In such cases, 
these fields are equivalent to and are intended to be 
covered by the broad terms "frame" and "video frame' 
as used in the frame processing of video signals. 
[0020] The present invention overcomes the prob- 
lem of progressive refresh related artifacts by conv 
mencing processing of a frame to be inserted prior to 
the insertion time. First, the video to be inserted is 
decompressed sufficiently prior to the insertion point to 
enable the recovery and recompression of a first frame 
to be inserted. As noted above, the "first frame to be 
inserted" is the first frame in the encoding order and not 
necessarily the first frame in the display order. Once this 
frame has been decompressed, it is recompressed as 
an intra-coded (I) frame. Such intra-coded frames are 
able to be reconstructed without refening to any other 
frame, and therefore are not subject to progressive 
refresh. After the recovered first frame is recompressed 
as an I frame, it is used as the first frame inserted at the 
insertion point 

[0021] As a first example. S1 may be a progres- 
sively refreshed compressed video stream whereas S2 
is not progressively refreshed. If a sequence is not pro- 
gressively refreshed then it is l-frame refreshed. In this, 
case, progressive refresh artifacts will occur In switching 
from S2 to Si . but not while going from SI to S2. There- 
tore, a switch can be made from Si to 82 in order to 
insert S2c in a conventional manner as is done for I- 
frame refreshed cases. However, the technique of the 
present invention must be used to avoid progressive 
refresh artifacts when switching back from S2c to SI . 
[0022] In accordance with the invention, when S2c 
is inserted, decoding off SI commences at some time dt 
eariier than the insertion point t2. In other words, decod- 
ing of SI wfll commwKe at time (t2-cfO. Upon recovery 
of a first frame (in encoding order) of S1 to be inserted 
at the termination off S2c. this frame or "picture" is rec- 
ompressed as all intra-coded blocte. and the reconrv 
pressed picture is inserted commencing at insertion 
point t2. The remainder of stream 81 can be the same 
as the original stream. 

[0023] In another example, both 81 and S2 are pro- 
gressively refreshed. In this case, when switching from 
stream SI to video segment S2c. decoding of S2c com- 
mences at some time dt' eariier tiian insertion t1 . i.e.. at 



(tl-cfr*). A recovered first frame (in encoding order) of 
82c is recompressed as all I blocks. This first frame is 
inserted as the first frame of tiie video segment S2c at 
insertion point t1. No special processing is required for 

5 the remainder of video segment 82c. When switching 
back from 82c to SI . the same technique used in the 
previous example is followed; i.e., tiie first frame (in 
encoding order) of stream 81 to be inserted at insertion 
poirt t2 is recovered and recompressed as all I Wocte 

10 for insertion at t2. 

[0024] FIG. 2 illustrates a special case referred to 
as "switching." In this case, a switch from one stream 
(e.g., a feed from a first camera) to another stream (e.g. . 
a feed from a second camera) is mada As illustrated in 

75 FIG. 2. stream 16 (stream S4) is aeated by switching 
81 to stream S2 at insertion point (or "switching point") 
t1 . In ttie event that SI is progressively refreshed but 82 
is not no special processing is required. However, in the 
case where S2 is progressively refreshed (regardless of 

20 whether 81 is progressively refreshed) decocfing of 81 
will be commenced at a time dt prior to the switching 
point t1 so ttiat a first frame thereof fin encoding order) 
can be recompressed as an I frame for use as the first 
frame when tiie streams are switched at switching poirrt 

25 t1- 

[0025] The times dt and dr are established by 
knowing how many frames it takes to complete the pro- 
gressive refresh of a picture. For example, if tiie pro- 
gressive refresh is completed within fifteen frames, dt 

30 and dt' will be a period of time at least equivalent to fif- 
teen frame times, and more typically will be greater ttian 
tills (e.g.. titirty frames) to be certain tiiat enough time is 
provided to decode and recompress tiie necessary f ffst 
frame of tiie inserted video portion. 

35 [0026] It should now be appreciated tiiat tiie 
present invention avoids progressive refresh artifacts by 
starting to decode a main stream, which is progressively 
refreshed, several frames before an inserted video por- 
tion (e.g., a commercial) is over. At the time when the 

40 commercial is over, tfie decoded picture (e.g.. P picture) 
of ttie main stream is captured and recompressed as an 
I frame. The main stream is then switched-in commenc- 
ing witti ttiat I frame. The rest of ttie frames in tiie main 
stream can be progressively refreshed frames, ft is 

45 noted that tiiere may still be some artifacts due to tiie 
fact that the I frame is not tiie true reference frame used 
to encode tiie main stream. However, such artifacts are 
generally significanfly less objectionable than tiie pro- 
gressive refresh artifacts that wouM otheranse occur. 

50 [0027] In addition, it is also possible to store tiie 
commercial or ottier inserted video segment such tiiat 
only ttie first frame of ttie segment is an I frame and tiie 
rest of the frames are progressively refreshed. In ttiis 
manner, a progressively refreshed commercial can be 

55 used wittiout detrimental progressive refreshed artifacts 
at the time tiie conrmercial is switched in to tiie video 
stream. 

[0028] Although the invention has been described 
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in connection with various specific embodiments, it 
shouid be appreciated that numerous adaptations and 
modifications may be made thereto without departing 
from the scope of the invention as set forth in the claims. 

Claims 

1. A method for replacing compressed video from a 
first source with compressed video from a second 
source in a video data stream, comprising the steps 
of: 



2. 



3. 



A method in accordance with claim 1 wherein the 
replacement of the first source video with the sec- 
ond source video commences with the insertion of 
said I frame at a pred^ined insertion point in said 
video data stream. 

A method in accordance with claim 2 wherein said 
insertion point Indicates the commencement of a 
commercial break in said video stream, and said 
second source provides a commercial for insertion 
into said commercial break. 

A method in accordance with claim 3 wherein at the 
conclusion of said commercial break, vkleo from 
said first source recommences at a second inser- 
tion point in said data stream, comprising the fur- 
ther steps of: 

determining if the vdeo from said first source is 
progressively refreshed video, and if so: 

(a) decompressing the video from said first 
source sufficiently prior to said second 
insertion point to enable the recovery of a 
first video frame from said first source to be 
inserted at said second insertion point; 

(b) recompressing the recovered first video 
frame from said first source to produce an 
intra-coded (I) frame: and 



10 



determining if the video from sakj second 
source is progressively refreshed vkleo. and if 

so: IS 



(a) decompressing the vWeo from said 
second source sufficiently prior to the 
replacement of the first source video to 
enable the recovery of a first replacement 
video frame from saxd second source 
before replacement commences; 

(b) recompressing the recovered first 
replacement video frame as an intra-coded 
(I) frame; and 

(c) commencing the replacement of the 
first source video with the second source 
video using sakJ I frame. 
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(c) commencing the insertion of the video 
from said first source at sakJ second inser- 
tion point using the I frame produced from 
said first source. 

. A method for replacing vkleo from a first source 
with substitute progressively refreshed compressed 
video In a vkleo data stream, comprising the steps 
of: 

storing the substitute video with a first frame 
thereof encoded as an intra-coded (I) frame; 
and 

commencing the insertion of the substitute 
video into said vkleo data stream using said I 
frame as the first inserted frame. 

A method In accordance with claim 5 wherein the 
insertion of the sul)stitute vkleo commences with 
the insertion of sakl I frame at a predefined inser- 
tion point in said vkleo data stream. 

A method in accordance with claim 6 wherein the 
compressed video from sakl first source is progres- 
sively refreshed and is to be recommenced at a 
second insertion point in sakl data stream following 
said first Insertion point sakl method comprising 
the further steps of: 

(a) decompressing the video from sakl first 
source suffidently prior to said second inser- 
tion point to enable the recovery and recom- 
pression of a first video frame from said first 
source to be inserted at sakl second insertion 
point; 

(b) recompressing the recovered first vkleo 
frame from sakl first source to produce an intra- 
coded (q frame; and 

(c) commencing the insertk)n of the vkleo from 
sakl first source at said second insertion point 
using the I frame produced from sakl first 
source. 

A method in accordance with daim 7 wherein sakl 
first and second insertion points define a commer- 
dal break in sakl video stream, and sakl sut)stitute 
video is a commercial for insertion into sakl com- 
mercial break. 
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(54) Splicing of video data in progressively refreshed video streams 



(57) A method is provided for replacing connpressed 
video from a first source with video from a second 
source at a predefined insertion point in a video data 
stream. If the video from the second source is progres- 
sively refreshed video, it is decompressed sufficiently 
prior to the insertion point to enable the recovery and 
recompression of a first video frame (in encoding order) 
from the second source to be inserted at the insertion 
point. The recovered first video frame is then recom- 



pressed as an intra-coded (I) frame. The insertion of the 
video from the second source is then commenced at the 
insertion point using the I frame. The insertion point can, 
for example, indicate the commencement of a connmer- 
clal break in the video stream, in which case the second 
source provides a commercial for insertion into the com- 
mercial brealc. Where the commercial is stored in mem- 
ory, it can be stored with the first frame thereof (in en- 
coding order) as an I frame. 
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